The development of software products such as those from IBM typically follows a process where the development team is responsible for the development and unit tests of the product, while a specialized testing team will be in charge of other verification tests, and problems found during the verification tests will be returned to be solved by the development team. A unit test is a white box test, the purpose of which is to test the code logics, while a verification test is a black box test, and is used to verify the functions and behaviors of the software.
During the verification tests, many runtime problems may be found, such as unexpected exceptions and incorrect or abnormal program behaviors. Since the testing team and the development team are independent of each other, and the testing environment and the development environment are not identical either, problems found during the verification tests typically can only be fed back to the development team through textual descriptions of the testers or the log information of the software itself. Because the testing environment lacks debugging tools, and the problem descriptions and log information cannot be directly used for debugging, the development team has to recreate a testing environment according to the problem descriptions for debugging, which is a very tedious and inefficient process. And sometimes, it is very difficult to reproduce the runtime problems and find their causes.
Moreover, when the development team has corrected the currently found problems or developed a new version of the software, since new code has been added, the testing team must still ensure that previously found and corrected problems remains being correct, without being affected by the current modifications, that is, to, perform the so-called regression tests. Although the testing team must perform regression tests, if the development team is able to ensure the correctness of the regression tests in advance, then the testing team needs only to perform a verification test, thus undoubtedly improving the test efficiency. Since the development team's testing tools are usually unit test cases, it is necessary for the development team to develop a unit test case for each found problem, which can verify whether it is corrected. But due to the reason of the previous problem, the debugging work being already complex, developing unit test cases based on problems is also a very tedious process, and is often omitted by the development team.
Current automatic testing tools, such as Rational Robot and WinRunner, etc., are only oriented to testing teams and testing processes, and its starting point is to substitute for the testers, repetitive actions. They focus on reproducing runtime problems via recording and playing back the end user's actions, such as keystrokes and mouse clicks, rather than capturing internal programmatic behaviors. They generate scripts to simulate the inputs from the end user or from the runtime environment. It is difficult to determine how many user interactions lave to be recorded in order to reproduce the problem. And the execution time required to reproduce the problem may not be optimal because unrelated activities may be contained in the playback script.
Unit test is now widely used in software development process. Developers can find and eliminate (defects quickly and simply by executing unit test suite before the program is delivered to the verification test. Generally speaking, the cost of unit test is much lower than the verification test.
There are also at present several technologies and pre-ducts that can help developers to generate unit test cases such as JTest. They generate unit test cases based on the static analysis of the source code rather than the runtime context. However, static analysis alone cannot determine the context of how the data is used. And it cannot make use of the runtime context and state to create test cases which can reproduce runtime problems. As such, none of them can link the testing process and development process so as to solve the above problems.
Therefore, if it is possible to automatically generate unit test cases, the above two problems will be solved effectively: on the one hand, the development team can directly use these unit test cases for debugging, saving the work of recreating the testing environment; on the other hand, the development team can add these test cases into the unit test process, to ensure the correctness of the regression test.
The U.S. Pat. No. 6,738,778, entitled “Method and Apparatus for Monitoring the Execution of a Program”, discloses a tracking and debugging tool, which can be invoked in a program, and can automatically collect and print program running information (e.g., which line of which class the program is currently running in). However, this patent only prints the tracking information for debugging, rather than generating unit test cases. In addition, it cannot actively find errors and exceptions in the execution of the program, and thereby activating the generation of unit test cases.
The U.S. Pat. No. 6,067,639, entitled “Method for Integrating Automated Software Testing with software Development” discloses an automatic software testing method that needs to be integrated in the development process. The method requires the software developers to develop some test operation objects while in the development process, each of which test operation objects implements invoking a certain operation on the program. Thus, in testing, the software testers can write automatic test programs or scripts, to automate the software testing by invoking the test operation objects in the program under test. Since these objects provide a uniform interface, the testers need not develop different automatic test programs or scripts for different testing environments. However, this patent requires the software developers to develop test operations objects based on a uniform interface, which in fact amounts to developing test cases, rather than automatically generating unit test cases. Moreover, it requires the developers to develop test operations objects while at the time of developing the program.
From the foregoing, it can be seen that there is apparently a need in the art for a method and system for automatically generating unit test eases based on runtime problems.